This projects uses deep neural networks and convolutional neural networks to clone driving behavior.
Built in Keras, the model is trained on sequence of images and steering angles collected by driving/steering the car manually in a simulator.
The model will predict a steering angle which is used to drive an autonomous vehicle.
Training a neural network to build a self driving car is no easy task. Building a machine to mimic human driving requires gathering lot of data parameters. A simple convolutional neural network which has the ability to remember basic patterns in a sequence of high quality images will have more than a million parameters, let alone a network which generalized well and operates perfect in all driving conditions.
In this project, the task is to build a neural network which will only predict the steering angle based on the road conditions. We have access to the a simulator which can be used to manually drive and collect data (images) using good driving behavior.
Simulator generates sequence of images using 3 cameras mounted on the hood of the car (Center, Left and Right). Steering angle is calculated using the Center camera.
import csv
import cv2
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
path = 'my_train_data/driving_log.csv'
driving_data = []
with open(path, 'r') as csvfile:
data = csv.reader(csvfile)
for row in data:
driving_data.append(row)
Let's look at a few images obtained using data collection.
We use 3 cameras on the car, the CENTER camera is always used to calculate the steer angle. For us to be able to use LEFT and RIGHT camera images a small correction factor of 0.2 is added to the center steer angle and tag the result to left camera. Similarly same factor of 0.2 is substracted from the center steer angle and tag the result to right camera. This factor teaches the network to steer back to the center if vehicle is drifting towards the sides of the road.
This correction is very helpful when for example the vehicle sees a left curve, it tells the network to steer a little hard to the right to stay on course.
random_idxs = random.sample(range(len(driving_data)), 3)
f, axarr = plt.subplots(3,3, figsize=(50,50))
x_axis = 0
for idx in random_idxs:
row = driving_data[idx]
images = [mpimg.imread(row[0]),mpimg.imread(row[1]),mpimg.imread(row[2])]
steer = round(float(row[3]),2)
left_steer = round(steer + 0.2,2)
right_steer = round(steer - 0.2,2)
steer_angles = [left_steer, steer, right_steer]
image_names = ['CENTER','LEFT','RIGHT']
y_axis = 0
for i in [1,0,2]:
axarr[x_axis,y_axis].imshow(images[i])
axarr[x_axis,y_axis].set_title(image_names[i] + ' Steer Angle: '+ str(steer_angles[i]), fontsize=40)
y_axis += 1
f.subplots_adjust(hspace=0.2)
x_axis += 1
plt.show()
Deep artificial neural networks require a large corpus of training data in order to effectively learn, where collection of such training data is often expensive and laborious. Data augmentation overcomes this issue by artificially inflating the training set with label preserving transformations. Recently there has been extensive use of generic data augmentation to improve Convolutional Neural Network (CNN) task performance.
Improving Deep Learning using Generic Data Augmentation
Data Augmentation is also a regularization technique to generate new training images from existing ones to boost the size of the training set. This will not only help the model learn better, but will also reduce over-fitting. The idea is to randomly adjust brightness, rotate/flip, vertical/horizontal shift every picture in the training set and add them back to the training set. This forces the model to be more tolerant to position and orientation of the key pixels in the image.
Using data from 3 cameras with correction factors applied to steering angles is our first step in Data augmentation, which we already did in previous step.
In this step of augmentation, adjust the brightness of the images by selecting a random intensity level for each image.
Intensity level is selected at random to simulate different sunlight conditions in a day. A darker intensity can be as close to driving at night.
Open CV reads images in BGR format. We first conver images from BGR to HSV(Hue-Saturation-Value) and randomly alter V value to change brightenss and finally convert the image back to RGB.
Drive.py in autonomous mode also gets the images from simulator using PIL image library which reads in RGB format.
Here are few examples of images after adjusting the brightness.
import numpy as np
def random_brightness_adjust(img):
new_img = cv2.cvtColor(img, cv2.COLOR_RGB2HSV)
brightness_level = (0.2 * np.random.uniform()) + 0.4
new_img[:,:,2] = new_img[:,:,2] * brightness_level
new_img = cv2.cvtColor(new_img, cv2.COLOR_HSV2RGB)
return new_img
import random
random_idxs = random.sample(range(len(driving_data)), 15)
images = []
for idx in random_idxs:
row = driving_data[idx]
clr = random.sample(range(3), 1)[0]
adj_image = random_brightness_adjust(mpimg.imread(row[clr]))
images.append(adj_image)
f, axarr = plt.subplots(5,3, figsize=(75,75))
x_axis = 0
y_axis = 0
for i in range(len(images)):
if i!=0 and i%3 == 0:
x_axis += 1
y_axis = 0
axarr[x_axis,y_axis].imshow(images[i])
y_axis += 1
#f.subplots_adjust(hspace=0 )
plt.show()